SpaceNet: Make Free Space for Continual Learning

The continual learning (CL) paradigm aims to enable neural networks learn tasks continually in a sequential fashion. fundamental challenge this is catastrophic forgetting previously learned when the model optimized for new task, especially their data not accessible. Current architectural-based methods aim at alleviating problem but expense of expanding capacity model. Regularization-based maintain fixed capacity; however, previous studies showed huge performance degradation these task identity available during inference (e.g. class incremental scenario). In work, we propose novel method referred as SpaceNet scenario where utilize intelligently. trains sparse deep from scratch an adaptive way that compresses connections each compact number neurons. training results representations reduce interference between tasks. Experimental show robustness our proposed against old and efficiency utilizing model, leaving space more be learned. particular, tested on well-known benchmarks CL: split MNIST, Fashion-MNIST, CIFAR-10/100, it outperforms regularization-based by big gap. Moreover, achieves better than without expansion achieved comparable with rehearsal-based methods, while offering memory reduction.